COLOR AS A VISUAL CUE IN SPEECH-ENABLED APPLICATIONS 

BACKGROUND OF THE INVENTION 

1. FIELD OF INVENTION 

[0001] The invention relates generally to speech enablement of 
software/hardware applications, and more specifically to using color as a visual 
cue to speech-enable a user interface to an information display. 

2. ART BACKGROUND 

[0002] A computer software program can be configured to interact with 
hardware, including a microphone, making the software program or application 
responsive to speech. Such a configuration is referred to as being speech- 
enabled. Currently, these applications rely on the user remembering the 
commands that will trigger a response to speech. Users may refer to the 
application's manual or help files to learn what is and is not speech-enabled. 
Further reference to the manual is required to learn the particular commands that 
will trigger responses to speech. Problems with this methodology arise. The 
user's memory is taxed as the application grows in size beyond a small number 
of commands. An application's full potential may not be realized if the user 
forgets commands that are speech-enabled. The particular command phrase 
that must be spoken is not evident in these existing applications; reference to the 
application's manual or other screens is necessary to learn the required phrase. 
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[0003] What is needed is a way of knowing what speech commands are 
available to the user within the application and more particularly within the 
application screen without taxing the user's memory or requiring the user to refer 
back to the application's manual or help screens. 
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BRIEF DESCRIPTION OF THE DRAWINGS 


[0004] The file of this patent contains at least one drawing executed in 
color. Copies of this patent with color drawing(s) will be provided by the Patent 
and Trademark Office upon request and payment of the necessary fee. The 
present invention is illustrated by way of example and is not limited in the figures 
of the accompanying drawings, in which like references indicate similar elements. 

[0005] Figure 1 illustrates a relationship between graphical characters, 
graphical commands, color, and a process within a speech-enabled application. 

[0006] Figure 1a is a first screen of a speech-enabled application using 
the color blue to indicate which commands are speech-enabled. 

[0007] Figure 2 is a second screen of the speech-enabled application 
using color as a navigational aid within the speech-enabled application. 

[0008] Figure 3 illustrates the use of what is implied by a graphical 
character using "i." 

[0009] Figure 4 illustrates dynamic speech hot keys. 
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DETAILED DESCRIPTION 

[001 0] In the following detailed description of embodiments of the 
invention, reference is made to the accompanying drawings in which like 
references indicate similar elements, and in which is shown by way of illustration, 
specific embodiments in which the invention may be practiced. These 
embodiments are described in sufficient detail to enable those skilled in the art to 
practice the invention. The following detailed description is, therefore, not to be 
taken in a limiting sense, and the scope of the invention is defined only by the 
appended claims. 

[001 1] Color is used as a visual cue to inform a user as to which 
commands are speech-enabled within an application or an application screen. 
Blue is used to illustrate the invention within the context of the figures presented 
herein. However, another color could be used to indicate speech-enabled 
commands, such as the color green. The present invention is not limited by the 
choice of color to indicate which commands are speech-enabled. 

[0012] Within this description, reference will be made to "graphical 
character" and "graphical command." Graphical character has broad meaning, 
encompassing any text, numeral, icon, or marking that is colored either in whole 
or in part by the color chosen to represent speech enablement. Graphical 
command may have broader scope than graphical character. Graphical 
command is the spoken form of the graphical character and encompasses what 
is actually expressed by the graphical character or what is implied by the 
graphical character. Thus, multiple speech triggers are supported to provide 


Patent Application 


6 


Atty Docket NO.42390.P12154 


greater utility in the speech enablement process. Figure 1 illustrates a 
relationship 50 between graphical characters, graphical commands, color, non- 
colored graphical characters, and a process within a speech-enabled application. 
With reference to Figure 1 , graphical characters 52 are seen by the user and 
impart knowledge to the user that the graphical characters speech-enable a 
range of graphical commands. The user may speak explicit graphical commands 
54 or implied graphical commands 56, based on the content of the graphical 
characters 52 and associated non-colored characters to perform/execute/launch 
or trigger a speech-enabled process 58 from the speech-enabled application. 

[0013] As previously mentioned, the color blue is used in Figure 1a to 
indicate which commands are speech-enabled. With reference to Figure 1a, an 
information display 101 is illustrated containing a screen 100 of a speech- 
enabled application. The graphical characters colored blue on the screen 100 
are speech-enabled, whereas the graphical characters that are not colored blue 
are not speech-enabled. The speech-enabled application may consist of a 
plurality of screens. In one embodiment, speech-enabled commands may differ 
from a first screen to a second screen. In another embodiment, the speech- 
enabled commands may be the same from the first screen to the second screen. 
The present invention is not limited by the architecture of the speech-enabled 
application. Figure 1a illustrates the first screen 100 of the speech-enabled 
application using the color blue to indicate which commands are speech-enabled. 
With reference to Figure 1a, graphical character 102 "Back," when spoken as a 
graphical command, would cause the application to execute a process of cycling 
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back to a previous screen. Similarly graphical character 1 04 "Next," when 
spoken as a graphical command, would cause the application to execute a 
process of advancing to the next screen in a sequence. In these examples, the 
graphical characters are in the form of the words " Back" and "Next" which are 
displayed in the color to blue to indicate speech-enabled graphical commands 
that may be executed from the first screen 100 of the speech-enabled 
application. As previously described, what is implied by the graphical character 
may also be spoken as the graphical command. "Next screen" may be a suitable 
graphical command to cause the application to execute the process of advancing 
to the next screen in the sequence. The scope of permissible phrases that result 
in permissible implied graphical commands is a design parameter of the 
application and does not limit the present invention. 

[0014] In one embodiment, Icons may be used as shown with 1 22 (an icon 
of a keyboard) to represent the graphical character. Here, various graphical 
commands may be spoken that are implied by the graphical character 122, such 
as "Enter." A process of saving information may be indicated by icon 124, which 
shows a disk. A process of printing information from screen 100 of the speech- 
enabled application is possible and indicated by the presence of printer icon 126. 
Printing may be initiated by speaking what is implied by the graphical character 
126, such as an implied graphical command "print." 

[001 5] The color of choice to indicate speech-enabled commands should 
be used, either in whole or in part, in association with a bounded region of the 
screen to indicate that the graphical character(s) corresponds to graphical 
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command(s) that perform/execute/launch or trigger a speech-enabled process 
from the application. For example, icons 122, 124, and 126 are colored, at least 
in part, blue consistent with the description presented herein. A second region of 
the information display contains elements (114 and 116) that do not appear in 
blue, either in whole or in part, and are not directly associated with speech- 
enabled commands. These terms are part of the application but are not explicitly 
connected with speech-enabled commands and processes by themselves. For 
example, speaking "Keyboard" will not perform/execute/launch or trigger a 
speech-enabled process from the application. However, these terms may be 
used in association with a graphical character(s) to help convey to the user what 
is implied by the graphical character(s), resulting in a range of permissible 
implied graphical commands. For example, "patient name" 114 may be used in 
association with "Ann" 120 to imply the graphical command "patient Ann," or 
"patient Ann Dooley" which then performs/executes/launches or triggers a 
speech-enabled process from the application. 

[0016] Logical association of graphical characters may be used to 
associate a plurality of characters on screen 1 00 with each other to 
perform/execute/launch or trigger a common speech-enabled process from the 
application. For example, 1 1 8 "3" and 1 20 "Ann Dooley" may be associated 
together such that by speaking "3" or "Ann Dooley" the same process is 
performed/executed/launched or triggered from the application. Additionally, 
logical association of non-blue elements may be combined with blue elements to 
imply various graphical commands that will launch the process. For example, 
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1 14 may be combined with 1 1 8 to imply graphical command "Patient 3." An 
alternative combination could be 114 and 120 implying graphical command 
"Patient Ann Dooley." Many other implied graphical commands are possible 
within the teaching of the present invention using color as described herein. 

[001 7] Graphical characters such as the question mark "?" at 1 08 may 
imply the graphical command "help." Multiple graphical commands may be used 
to indicate the same process as shown at 1 10; here "Med Ref and "Mr" are used 
to denote "medical reference." A menu is indicated at 106 with the graphical 
command "menu." 

[0018] Another screen of the speech-enabled application may be 
displayed by speaking the graphical command at 104 "next" resulting in screen 
200 being displayed on the information display 101, as shown in Figure 2. 
Figure 2 is a second screen of a speech-enabled application illustrating some 
speech-enabled graphical characters that are different from the first screen and 
some that are the same. Color may also be used as a navigational aid within the 
application as shown in Figure 2. The color of the graphical character "Next" at 
204 is no longer blue in screen 200, indicating that this formerly available 
graphical character (in screen 1 00 Figure 1 a) is no longer available from the 
present screen to be used as the graphical command to initiate the process that 
cycles forward to a next screen. This example illustrates using color as the 
navigational aid to indicate the end of a succession of screens that may be 
arranged in a path or tree structure. The graphical character "Back" at 102 is 
available as a navigational choice; in this way color is being used as the 


Patent Application 


10 


Atty Docket No.42390.P12154 


navigational aid to decide which way to proceed within the speech-enabled 
application. 

[0019] In one embodiment, new graphical commands are evident in 
screen 200. Here various associations of graphical characters, such as "1" at 
206 and "More" at 208 may be used to perform/execute/launch or trigger a 
speech-enabled process from the application. As described in conjunction with 
Figure 1a, non-blue elements may be associated with the graphical characters to 
form associations that imply graphical commands to perform/execute/launch or 
trigger a speech-enabled process from the application, such as using 202 and 
206 to speak the graphical command "Option 1." In addition, 204 may be 
combined with 208 to imply the graphical command "More Rxs for Ann" or 
"Option 1 for Ann. 

[0020] "In another embodiment, Figure 3 illustrates the use of what is 
implied by a graphical character using "i." Here, the graphical character "i" at 
302, when spoken as a graphical command would perform/execute/launch or 
trigger a speech-enabled process from the application. Alternatively, information 
is implied from the graphical character at 302. Therefore speaking the implied 
graphical command "information" or "info" would perform/execute/launch or 
trigger the same speech-enabled process from the application. Various 
combinations of a graphical character "patient" at 304 and non-blue elements at 
306, "Dooley, Ann (2-13-73)" may be logically associated to imply graphical 
commands to perform/execute/launch or trigger the same speech-enabled 
process from the application. 
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[0021] Another term for graphical character that will be used in this 
description is "speech hotkey." In some speech-enabled applications it may be 
convenient to express a region on the information display in the form or shape of 
a "key" or a "button" according to terms commonly used in the art. Therefore, 
speech hotkey may be used interchangeably with graphical character, no 
limitation is implied by the use of one term or the other term. Speech hotkeys 
may be configured to be dynamic. Figure 4 illustrates the use of dynamic 
speech hot keys. With reference to Figure 4, 402, 404, 406 and 408 represent 
speech hotkeys. The speech hotkeys change based on how much of a specific 
content a screen contains (e.g., how many different drug names are on the 
screen) and where that content is located on the screen (e.g., depending on 
where "Naproxen" is on the screen in relationship to the other drug names). For 
example, four drugs are shown in Figure 4 on screen 400 and are referenced 
with hotkeys, 402, 404, 406, and 408. If three drugs had been returned instead, 
only 402, 404, and 406 would be listed. Thus, the number of hotkeys is 
dynamically adjusted based on the results of a particular process executed within 
the application. 

[0022] In one embodiment, anytime a drug name appears on the page, a 
"MR" icon is placed next to it. Activating this icon, by speaking the graphical 
command associated with the icon will take the user to medical reference 
information for the associated drug. In a typical, non-speech-enabled 
application, the "MR" icons would all be the same because they perform the 
same action for their associated drugs and there is no need to uniquely 
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distinguish the meaning of the icons by where they are located on the page (i.e., 
which drug they are next to). The user would simply click on the desired icon 
with a stylus. In the context of the speech-enabled application explained herein, 
the user would speak the graphical command "MR" plus the number, required to 
make the combination unique, in order to access medical reference information 
for the associated drug. For example, if the user desired to go to medical 
reference information for Zyrtec, the user would say, "MR two." If the user 
desired to go to medical reference information for Premarin, the user would say, 
MR Three." And as described above, the user can also voice what is implied by 
the speech hotkey, which is "Med Ref 2" or "Medical Reference 2." Thus, speech 
hotkeys (graphical characters) may be used dynamically according to the results 
of the speech-enabled process within the speech-enabled application. 
[0023] It will be appreciated that the methods described in conjunction with 
the figures may be embodied in machine-executable instructions, e.g. software. 
The instructions can be used to cause a general-purpose or special-purpose 
processor that is programmed with the instructions to perform the operations 
described. Alternatively, the operations might be performed by specific hardware 
components that contain hardwired logic for performing the operations, or by any 
combination of programmed computer components and custom hardware 
components. The methods may be provided as a computer program product that 
may include a machine-readable medium having stored thereon instructions, 
which may be used to program a computer (or other electronic devices) to 
perform the methods. For the purposes of this specification, the terms "machine- 
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readable medium" shall be taken to include any medium that is capable of storing 
or encoding a sequence of instructions for execution by the machine and that 
cause the machine to perform any one of the methodologies of the present 
invention. The term "machine-readable medium" shall accordingly be taken to 
included, but not be limited to, solid-state memories, optical and magnetic disks, 
and carrier wave signals. Furthermore, it is common in the art to speak of 
software, in one form or another (e.g., program, procedure, process, application, 
module, logic...), as taking an action or causing a result. Such expressions are 
merely a shorthand way of saying that execution of the software by a computer 
causes the processor of the computer to perform an action or a produce a result. 
[0024] Thus, a novel method and apparatus to select a process by 
speaking a command both explicitly and impliedly by a user in a speech-enabled 
application is disclosed. Although the invention is described herein with 
reference to specific preferred embodiments, many modifications therein will 
readily occur to those of ordinary skill in the art. Accordingly, all such variations 
and modifications are included within the intended scope of the invention as 
defined by the following claims. 
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